Encode dep graph edges directly from the previous graph when promoting #122070

Zoxc · 2024-03-06T07:25:26Z

This encodes dep graph edges directly from the previous graph when promoting nodes from a previous session, avoiding allocations / copies.

~~Based on #122064 and #116375.~~

Benchmark	Before	After
Benchmark	Time	Time	%
🟣 clap:check:unchanged	0.4177s	0.4072s	💚 -2.52%
🟣 hyper:check:unchanged	0.1430s	0.1420s	-0.69%
🟣 regex:check:unchanged	0.3106s	0.3038s	💚 -2.19%
🟣 syn:check:unchanged	0.5823s	0.5688s	💚 -2.33%
🟣 syntex_syntax:check:unchanged	1.3992s	1.3692s	💚 -2.14%
Total	2.8528s	2.7910s	💚 -2.17%
Summary	1.0000s	0.9803s	💚 -1.97%

rustbot · 2024-03-06T07:25:34Z

r? @fmease

rustbot has assigned @fmease.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Zoxc · 2024-03-06T07:26:01Z

r? @cjgillot

cjgillot · 2024-03-08T23:33:21Z

@bors try @rust-timer queue

Encode dep graph edges directly from the previous graph when promoting This encodes dep graph edges directly from the previous graph when promoting nodes from a previous session, avoiding allocations / copies. Based on rust-lang#122064 and rust-lang#116375. <table><tr><td rowspan="2">Benchmark</td><td colspan="1">Before</th><td colspan="2">After</th></tr><tr><td align="right">Time</td><td align="right">Time</td><td align="right">%</th></tr><tr><td>🟣 clap:check:unchanged</td><td align="right">0.4177s</td><td align="right">0.4072s</td><td align="right">💚 -2.52%</td></tr><tr><td>🟣 hyper:check:unchanged</td><td align="right">0.1430s</td><td align="right">0.1420s</td><td align="right"> -0.69%</td></tr><tr><td>🟣 regex:check:unchanged</td><td align="right">0.3106s</td><td align="right">0.3038s</td><td align="right">💚 -2.19%</td></tr><tr><td>🟣 syn:check:unchanged</td><td align="right">0.5823s</td><td align="right">0.5688s</td><td align="right">💚 -2.33%</td></tr><tr><td>🟣 syntex_syntax:check:unchanged</td><td align="right">1.3992s</td><td align="right">1.3692s</td><td align="right">💚 -2.14%</td></tr><tr><td>Total</td><td align="right">2.8528s</td><td align="right">2.7910s</td><td align="right">💚 -2.17%</td></tr><tr><td>Summary</td><td align="right">1.0000s</td><td align="right">0.9803s</td><td align="right">💚 -1.97%</td></tr></table>

bors · 2024-03-08T23:34:32Z

⌛ Trying commit d9fe043 with merge ae110c4...

bors · 2024-03-09T01:03:19Z

☀️ Try build successful - checks-actions
Build commit: ae110c4 (ae110c4e0298a288fd51f34ac53ca2e19f205e2f)

rust-timer · 2024-03-09T07:27:17Z

Finished benchmarking commit (ae110c4): comparison URL.

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.6%	[0.2%, 1.2%]	25
Regressions ❌ (secondary)	0.8%	[0.7%, 0.9%]	4
Improvements ✅ (primary)	-0.7%	[-1.8%, -0.2%]	32
Improvements ✅ (secondary)	-1.1%	[-2.8%, -0.2%]	22
All ❌✅ (primary)	-0.2%	[-1.8%, 1.2%]	57

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	3.3%	[0.9%, 6.1%]	4
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-3.2%	[-4.0%, -1.9%]	6
All ❌✅ (primary)	-	-	0

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.7%	[2.3%, 3.4%]	3
Improvements ✅ (primary)	-2.4%	[-7.0%, -1.4%]	53
Improvements ✅ (secondary)	-3.0%	[-6.1%, -1.2%]	20
All ❌✅ (primary)	-2.4%	[-7.0%, -1.4%]	53

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 649.968s -> 649.736s (-0.04%)
Artifact size: 172.54 MiB -> 172.56 MiB (0.01%)

cjgillot

Could you provide an explanation where the gains come from? I suspected we remove the need to decode the edges, but that's not it.

compiler/rustc_query_system/src/dep_graph/serialized.rs

cjgillot · 2024-03-09T22:19:20Z

compiler/rustc_query_system/src/dep_graph/graph.rs

+                self.record_edge(
+                    dep_node_index,
+                    prev_graph.index_to_node(prev_index),
+                    prev_graph.fingerprint_by_index(prev_index),


Should we keep the let-bindings for those two? They are computed by promote IIUC.

I'd prefer to keep them close to their use in promote.

cjgillot · 2024-03-09T22:21:44Z

compiler/rustc_query_system/src/dep_graph/serialized.rs

-            if let Some(record_graph) = &mut record_graph.try_lock() {
-                record_graph.push(index, node.node, &node.edges);
-            }
+            let edges = edges(self);


Move this call inside the closure?

I wanted the closure construction to be able to be optimized out.

Ok. Could you add a comment to justify the outline?

cjgillot · 2024-03-10T13:03:10Z

compiler/rustc_query_system/src/dep_graph/serialized.rs

+        node: DepNode,
+        fingerprint: Fingerprint,
+        prev_index: SerializedDepNodeIndex,
+        prev_index_to_index: &mut IndexVec<SerializedDepNodeIndex, Option<DepNodeIndex>>,


Why &mut?

I was thinking about uniqueness, but it doesn't help here. I've changed it to &.

Zoxc · 2024-03-10T13:22:27Z

compiler/rustc_query_system/src/dep_graph/graph.rs

-                let edges = prev_graph
-                    .edge_targets_from(prev_index)
-                    .map(|i| prev_index_to_index[i].unwrap())
-                    .collect();


The PR avoids this copy and allocation into EdgesVec.

Zoxc · 2024-03-13T10:23:07Z

@rustbot ready

cjgillot · 2024-03-23T17:35:19Z

@bors r+

bors · 2024-03-23T17:35:21Z

📌 Commit 4704852 has been approved by cjgillot

It is now in the queue for this repository.

cjgillot · 2024-03-23T17:40:52Z

compiler/rustc_query_system/src/dep_graph/serialized.rs

+    }
+
+    #[inline]
+    fn encode_promoted<D: Deps>(


Could you add a comment on this method, to explain the difference with encode?

cjgillot · 2024-03-23T17:41:23Z

compiler/rustc_query_system/src/dep_graph/serialized.rs

-            let stat = stats.entry(kind).or_insert(Stat { kind, node_counter: 0, edge_counter: 0 });
-            stat.node_counter += 1;
-            stat.edge_counter += edge_count as u64;
+            outline(move || {


Could you add a comment to justify the outline?

cjgillot · 2024-03-23T17:41:29Z

compiler/rustc_query_system/src/dep_graph/serialized.rs

-            if let Some(record_graph) = &mut record_graph.try_lock() {
-                record_graph.push(index, node.node, &node.edges);
-            }
+            let edges = edges(self);


Ok. Could you add a comment to justify the outline?

cjgillot · 2024-03-23T17:42:01Z

compiler/rustc_query_system/src/dep_graph/serialized.rs

+    }
+
+    #[inline]
+    fn promote_node(


Could you document this method? and the difference with encode_node?

cjgillot · 2024-03-23T17:42:44Z

@bors r-

Actually, could you add a few comments there and there?
r=me afterwards

Zoxc · 2024-04-05T09:05:27Z

@oli-obk Do you mind approving for me?

oli-obk · 2024-04-05T09:08:25Z

@bors r=cjgillot

bors · 2024-04-05T09:08:28Z

📌 Commit aa9c9a3 has been approved by cjgillot

It is now in the queue for this repository.

bors · 2024-04-05T11:11:21Z

⌛ Testing commit aa9c9a3 with merge 4563f70...

bors · 2024-04-05T13:16:00Z

☀️ Test successful - checks-actions
Approved by: cjgillot
Pushing 4563f70 to master...

rust-timer · 2024-04-05T14:33:44Z

Finished benchmarking commit (4563f70): comparison URL.

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Next Steps: If you can justify the regressions found in this perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please open an issue or create a new PR that fixes the regressions, add a comment linking to the newly created issue or PR, and then add the perf-regression-triaged label to this PR.

@rustbot label: +perf-regression
cc @rust-lang/wg-compiler-performance

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.4%	[0.4%, 0.4%]	1
Improvements ✅ (primary)	-1.6%	[-3.1%, -0.2%]	113
Improvements ✅ (secondary)	-1.5%	[-3.6%, -0.3%]	37
All ❌✅ (primary)	-1.6%	[-3.1%, -0.2%]	113

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	4.1%	[4.1%, 4.1%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-2.9%	[-7.8%, -0.8%]	99
Improvements ✅ (secondary)	-3.3%	[-7.3%, -1.5%]	27
All ❌✅ (primary)	-2.9%	[-7.8%, -0.8%]	99

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 668.072s -> 669.208s (0.17%)
Artifact size: 317.99 MiB -> 318.05 MiB (0.02%)

Kobzol · 2024-04-05T14:49:49Z

The improvements obviously overweigh the single regression.

@rustbot label: +perf-regression-triaged

rustbot assigned fmease Mar 6, 2024

rustbot assigned cjgillot and unassigned fmease Mar 6, 2024

This comment has been minimized.

Sign in to view

Zoxc force-pushed the dep-edges-from-previous branch from de02827 to 34a3025 Compare March 6, 2024 08:06

This comment has been minimized.

Sign in to view

Zoxc force-pushed the dep-edges-from-previous branch from 34a3025 to d9fe043 Compare March 6, 2024 08:35

Zoxc mentioned this pull request Mar 7, 2024

Represent diagnostic side effects as dep nodes #122156

Merged

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Mar 8, 2024

This comment has been minimized.

Sign in to view

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels Mar 9, 2024

Zoxc force-pushed the dep-edges-from-previous branch from d9fe043 to bb9f99c Compare March 10, 2024 08:53

Zoxc mentioned this pull request Mar 10, 2024

Create the previous dep graph index on a background thread #116375

Draft

cjgillot reviewed Mar 10, 2024

View reviewed changes

cjgillot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 10, 2024

Zoxc commented Mar 10, 2024

View reviewed changes

Zoxc force-pushed the dep-edges-from-previous branch from bb9f99c to 4704852 Compare March 11, 2024 02:28

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Mar 13, 2024

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 23, 2024

cjgillot reviewed Mar 23, 2024

View reviewed changes

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Mar 23, 2024

Zoxc added 2 commits March 23, 2024 20:03

Encode dep graph edges directly from the previous graph when promoting

6119763

Add some comments and do some renames

aa9c9a3

Zoxc force-pushed the dep-edges-from-previous branch from 4704852 to aa9c9a3 Compare March 23, 2024 19:24

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Apr 5, 2024

bors added the merged-by-bors This PR was explicitly merged by bors. label Apr 5, 2024

bors merged commit 4563f70 into rust-lang:master Apr 5, 2024

rustbot added this to the 1.79.0 milestone Apr 5, 2024

rustbot added the perf-regression Performance regression. label Apr 5, 2024

rustbot added the perf-regression-triaged The performance regression has been triaged. label Apr 5, 2024

Uh oh!

Encode dep graph edges directly from the previous graph when promoting #122070

Encode dep graph edges directly from the previous graph when promoting #122070

Uh oh!

Conversation

Zoxc commented Mar 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Mar 6, 2024

Uh oh!

Zoxc commented Mar 6, 2024

Uh oh!

This comment has been minimized.

This comment has been minimized.

cjgillot commented Mar 8, 2024

Uh oh!

This comment has been minimized.

bors commented Mar 8, 2024

Uh oh!

bors commented Mar 9, 2024

Uh oh!

This comment has been minimized.

rust-timer commented Mar 9, 2024

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Uh oh!

cjgillot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Zoxc commented Mar 13, 2024

Uh oh!

cjgillot commented Mar 23, 2024

Uh oh!

bors commented Mar 23, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cjgillot commented Mar 23, 2024

Uh oh!

Zoxc commented Apr 5, 2024

Uh oh!

oli-obk commented Apr 5, 2024

Uh oh!

bors commented Apr 5, 2024

Uh oh!

bors commented Apr 5, 2024

Uh oh!

bors commented Apr 5, 2024

Uh oh!

rust-timer commented Apr 5, 2024

Overall result: ❌✅ regressions and improvements - ACTION NEEDED

Uh oh!

Kobzol commented Apr 5, 2024

Uh oh!

Reviewers

Assignees

Labels

Zoxc commented Mar 6, 2024 •

edited

Loading